System with emulator movement tracking for controlling medical devices

ABSTRACT

The systems and methods disclosed herein are directed to robotically controlling a medical device to utilize manual skills and techniques developed by surgeons. The system can include an emulator representing a medical device. The system can include at least one detector configured to track the emulator. The system can also include an imaging device configured to track the medical device. The system may be configured to move the medical device to reduce an alignment offset between the location of the emulator and the location of the medical device, to move the imaging device based on the translational movement of the emulator, and/or to move the medical device based on data indicative of an orientation of the emulator.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit as a continuation of U.S. patentapplication Ser. No. 15/657,051, filed Jul. 21, 2017, which claims thebenefit of U.S. Provisional Application No. 62/365,308, filed Jul. 21,2016, which are hereby incorporated by reference in their entirety.

BACKGROUND

The use of robotic medical technologies presents a number of advantagesover traditional, manual medical procedures (e.g., surgery). Roboticsurgeries allow for higher degree of precision, control, and accessamong many other advantages. Despite these advantages and recentimprovements in the technology, many existing robotic surgical platformsare limited by their user interfaces. A great majority of surgicalrobotic interfaces comprise joysticks or other mechanical devicesmounted to an instrument that is manipulated by an operator (e.g., thesurgeon) to control the tools performing surgery on the patient. Most ofthese interfaces are not intuitive nor are they designed to mimic thesurgical motions and skills that surgeons have spent a multitude ofhours training and honing. Generally, the direct manipulation of roboticelements via the mechanical interface requires a separate trainingregimen forcing the physician to learn new skills rather than employingtechniques developed over many years.

An additional problem to robotic joystick systems is that the hapticfeedback of a robotic system is not a similar representation of thehaptic feedback received from traditional surgical tools. In a typicalprocedure, surgical tools have a specific weight, feel, and ease ofmotion. In robotic procedures, the joystick system is engineered to tryand mimic these characteristics but prove lacking in many instances,e.g. a mounted joystick with actuators is not able to have the same easeof movement and degree of freedom as a hand holding a scalpel. Thisinconsistency forces the operator to develop a separate set of responsesfor the haptic feedback cues received from the joystick system,complicating the transition to a robotic system.

For these reasons, it would be desirable to provide additional andalternative user interfaces for the performance of robotically assistedmedical, surgical, and diagnostic procedures. Such interfaces andmethods for their use should allow physicians to manipulate tools in amanner that more closely mimics the use of conventional tools innon-robotic medical or surgical procedures.

SUMMARY

The systems, methods and devices of this disclosure each have severalinnovative aspects, no single one of which is solely responsible for thedesirable attributes disclosed herein.

One aspect relates to a system, comprising: an emulator configured to beheld and operated in a free working space (FWS), the emulatorrepresenting a medical device at a target site; at least one detectorconfigured to track the emulator within the FWS; at least onecomputer-readable memory having stored thereon executable instructions;and at least one processor in communication with the at least onecomputer-readable memory and configured to execute the instructions tocause the system to: determine an alignment offset between a location ofthe emulator and a location of the medical device; during a medicalprocedure, determine a first movement amount based on a signal from theat least one detector indicative of a first movement of the emulatorwithin the FWS; adjust the first movement amount by a first adjustmentvalue; and generate instructions to move the medical device based on theadjusted first movement amount, wherein movement of the medical deviceby the adjusted first movement amount reduces the alignment offsetbetween the location of the emulator and the location of the medicaldevice.

In some implementations, the at least one processor is configured toexecute the instructions to cause the system to: during the medicalprocedure, determine a second movement amount based on a signal from theat least one detector indicative of a second movement of the emulatorwithin the FWS; adjust the second movement amount by a second adjustmentvalue; and generate instructions to move the medical device based on theadjusted second movement amount, wherein movement of the medical deviceby the adjusted second movement amount reduces the alignment offsetbetween the location of the emulator and the location of the medicaldevice.

In some implementations, the movement of the medical device by theadjusted first and second movement amounts eliminates the alignmentoffset between the location of the emulator and the location of themedical device.

Another aspect relates to a system, comprising: an emulator configuredto be held and operated in a FWS, the emulator representing a medicaldevice at a target site; at least one detector configured to track theemulator within the FWS; an imaging device configured to track themedical device at the target site; at least one computer-readable memoryhaving stored thereon executable instructions; and at least oneprocessor in communication with the at least one computer-readablememory and configured to execute the instructions to cause the systemto: receive a signal from the at least one detector indicative of atranslational movement of the emulator within the FWS; and generateinstructions, based on the translational movement of the emulator, tomove the imaging device within a plane defined by pitch and yaw axes ofthe imaging device.

In some implementations, the translational movement of the emulator doesnot result in a translational movement of the medical device. In someimplementations, the at least one processor is configured to execute theinstructions to cause the system to: receive a signal from the at leastone detector indicative of a rotational movement of the emulator withinthe FWS; and generate instructions, based on the rotational movement ofthe emulator, to rotate the medical device along a roll axis of themedical device. In some implementations, the rotational movement of theemulator does not result in a rotational movement of the imaging device.

Yet another aspect relates to a system, comprising: an emulatorrepresenting a medical device at a target site; a first set of one ormore detectors configured to track the emulator; a second set of one ormore detectors configured to track the medical device at the targetsite; at least one computer-readable memory having stored thereonexecutable instructions; and at least one processor in communicationwith the at least one computer-readable memory and configured to executethe instructions to cause the system to: receive, from the first set ofone or more detectors, first data indicative of at least an orientationof the emulator, the first data comprising roll data, pitch data, andyaw data of the emulator; generate, based on a clutched user input,instructions to move the medical device based on the first datadiscounting the roll data of the emulator; and cause the medical deviceto move based on the instructions.

In some implementations, the emulator is configured to be held andoperated in a FWS; and the first set of one or more detectors isconfigured to track motion of the emulator in the FWS. In someimplementations, the emulator comprises a mechanical emulator; and thefirst set of one or more detectors is configured to track mechanicalmovement of the emulator.

In some implementations, the discounting of the roll data of theemulator is based on decoupling a roll axis of the emulator from yaw andpitch axes of the emulator. In some implementations, the discounting ofthe roll data of the emulator is based on decoupling an absolute rollangle of the medical device from an absolute roll angle of the emulator.

In some implementations, the emulator is symmetric with respect to aroll axis of the emulator. In other implementations, the emulator isasymmetric with respect to a roll axis of the emulator.

In some implementations, the movement of the medical device based on theinstructions facilitates adjustment of a roll axis of the emulator withrespect to a roll axis of the medical device. In some implementations,the at least one processor is configured to execute the instructions tocause the system to receive, from the second set of one or moredetectors, second data indicative of an orientation of the medicaldevice at the target site, the second data comprising roll data, pitchdata, and yaw data of the medical device; and the alignment of therespective roll axes of the emulator and the medical device is based onthe pitch and yaw data of the emulator and the pitch and yaw data of themedical device.

In some implementations, the at least one processor is configured toexecute the instructions to cause the system to: receive, from the firstset of one or more detectors, third data indicative of a translationalmovement of the emulator; receive, from the second set of one or moredetectors, fourth data indicative of a position of the medical device atthe target site; and generate instructions to move the medical devicebased on the third and fourth data.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed aspects will hereinafter be described in conjunction withthe appended drawings and appendices, provided to illustrate and not tolimit the disclosed aspects, wherein like designations denote likeelements.

FIG. 1 illustrates an embodiment of a master/slave surgical tool systemfor minimally invasive surgery, the master device employing joysticksfor manipulating the slave surgical tools.

FIG. 2 illustrates an embodiment of a three-dimensional motion trackingsystem.

FIG. 3A illustrates an embodiment of master device movement within thefree working space while the master/slave surgical system is in activemode.

FIG. 3B illustrates an embodiment of slave device effector movementshown on a video display while the master/slave surgical system is inactive mode.

FIG. 3C illustrates an embodiment of master device movement within thefree working space while the master/slave surgical system is in cameramode.

FIG. 3D illustrates an embodiment of slave device camera movementchanging the field of view on video display while the master/slavesurgical system is in camera mode.

FIG. 3E illustrates an embodiment of master device movement within thefree working space while the master/slave surgical system is inalignment mode.

FIG. 3F illustrates an embodiment of slave device effector inactivity ona video display while the master/slave surgical system is in alignmentmode.

FIG. 4A illustrates representative effectors and emulators within a freeworking space, according to one embodiment.

FIG. 4B illustrates a system for providing haptic feedback duringalignment of a master/slave surgical system, according to oneembodiment.

FIG. 5A illustrates the free working space of a master/slave surgicalsystem operating in active mode without relative misalignment and withthe emulators within the limited working area, according to oneembodiment.

FIG. 5B illustrates the video display during active mode operation whenthe effectors remain within the limited working area, according to oneembodiment.

FIG. 5C illustrates a situation within the free working space of amaster/slave surgical system triggering alignment mode operation due tomotion of the emulators near the boundaries of the representativelimited working area causing relative misalignment, according to oneembodiment.

FIG. 5D illustrates the video display during alignment mode operationwhen the effectors are at the boundaries of the limited working area,according to one embodiment.

FIG. 5E illustrates a situation within the free working space of amaster/slave surgical system reentering active mode operation due torealignment of the position of the emulators and the position of therepresentative effectors, according to one embodiment.

FIG. 5F illustrates the video display during active mode operation dueto realignment of the position of the emulators and the position of therepresentative effectors, according to one embodiment.

FIG. 6 illustrates an emulator designed for grip detection, according toone embodiment.

FIG. 7A illustrates translational movement of the emulators within thefree working space for control of the surgical effectors at the surgicalsite while the master/slave surgical system is operating in active mode,according to one embodiment.

FIG. 7B illustrates translational movement of the effectors resultingfrom translational movement of the emulators displayed on the videoscreen while the master/slave surgical system is operating in activemode, according to one embodiment.

FIG. 7C illustrates translational movement of the emulators within thefree working space for control of the camera at the surgical site whilethe master/slave surgical system is operating in camera mode, accordingto one embodiment.

FIG. 7D illustrates translational movement of the camera resulting fromtranslational movement of the emulators displayed on the video screenwhile the master/slave surgical system is operating in camera mode,according to one embodiment.

FIG. 7E illustrates rotational movement of the emulators within the freeworking space for control of the effectors at the surgical site whilethe master/slave surgical system is operating in camera mode, accordingto one embodiment.

FIG. 7F illustrates rotational movement of the effectors resulting fromrotational movement of the emulators displayed on the video screen whilethe master/slave surgical system is operating in camera mode, accordingto one embodiment.

FIG. 8A is an illustration of the emulators and the representativeeffectors during the first iteration of the ratcheting process,according to one embodiment.

FIG. 8B is an illustration of the effectors during the first iterationof the ratcheting process, according to one embodiment.

FIG. 8C is an illustration of the emulators and the representativeeffectors during the second iteration of the ratcheting process,according to one embodiment.

FIG. 8D is an illustration of the effectors during the second iterationof the ratcheting process, according to one embodiment.

FIG. 8E is an illustration of the emulators and the representativeeffectors during the third iteration of the ratcheting process,according to one embodiment.

FIG. 8F is an illustration of the effectors during the third iterationof the ratcheting process, according to one embodiment.

FIG. 9 illustrates a motion tracking system that self-monitorscalibration, according to one embodiment.

FIG. 10A illustrates a wearable surgical viewing system that translatesa large magnification distance to large surgical effector movements,according to one embodiment.

FIG. 10B illustrates a wearable surgical viewing system that translatesa small magnification distance to small surgical effector movements,according to one embodiment.

FIG. 11A illustrates a surgical viewing system with a changeablemagnification setting that translates small magnifications to largesurgical effector movements, according to one embodiment.

FIG. 11B illustrates a surgical viewing system with a changeablemagnification setting that translates large magnifications to smallsurgical effector movements, according to one embodiment.

FIGS. 12A-12F illustrate an emulator and a representative medical deviceduring a clutched roll process, according to one embodiment.

FIGS. 13A-13B illustrate an emulator, according to another embodiment.

FIG. 14 illustrates an emulator, according to yet another embodiment.

FIG. 15 illustrates an emulator, according to still another embodiment.

DETAILED DESCRIPTION

This description presents a method for robotically controlling medicaldevices (e.g., surgical effectors, tools, devices, or instruments) at atarget site (e.g., a surgical site) that utilizes the highly specializedmanual skills and techniques developed by surgeons over time.Specifically, a motion tracking system that directly translates themotion of hand(s) of an operator (e.g., a surgeon) in free space tomotions of a robotic surgical effector at the surgical site. The systemtracking, interpreting, and translating the position and actions of thesurgeon creates a more characteristic experience for the operator. Thus,the presented robotic surgical system yields a more natural transitiontowards advanced robotic techniques for surgical operations from thewidely used manual techniques.

Additionally, the system includes an emulator held by the operator andoperated in a free working space (FWS), the emulator representing acorresponding medical device at the target site. The emulator designclosely mimics the corresponding medical device at the target site inweight, range of motion, and functional abilities, e.g. the surgeonholding surgical scissor emulators is controlling surgical scissors atthe target site. The system may comprise at least one detectorconfigured to track the emulator within the FWS and/or at least onedetector (e.g., imaging device) configured to track the medical deviceat the target site. The detector may comprise one or more components ofelectromechanical systems such as, for example, encoders,potentiometers, linear variable differential transformers (LVDT), androtary variable differential transformers (RVDT).

The description below contains the following sections:

-   -   Section I: Describes an overview of a conventional master/slave        robotic surgical process.    -   Section II: Describes an example of a three-dimensional motion        tracking system for controlling slave surgical effectors at the        surgery site.    -   Section III: Describes the operative modes of a master/slave        surgical system.    -   Section IV: Describes a haptic response to indicate master/slave        alignment.    -   Section V: Describes a system for handling motion of the master        device that would create motion of the slave device outside of        its possible range of motion.    -   Section VI: Describes an emulator for detecting the operator's        grip on the emulators.    -   Section VII: Describes a system for controlling a slave side        camera with the emulators.    -   Section VIII: Describes a method for the automatic elimination        of master-slave alignment offsets.    -   Section IX: Describes a motion tracking system that        self-monitors the quality of its calibration.    -   Section X: Describes a system that adjusts the master/slave        motion ratio based on the view of the operator.    -   Section XI: Describes a system for clutched roll.        I. Master/Slave Surgical System

FIG. 1 is an illustration of a conventional master/slave roboticsurgical system 100, used to perform surgical operations. The systemcomprises a slave device 102 and a master device 104. The slave devicecomprises a surgical effector 106 that performs surgical procedures at asurgical site 108. The movements of the surgical effectors are areproduction of movement representative of the surgical procedureperformed by the operator 110 at the master device. The master/slavesurgical system operates in one of three modes: active, alignment, orcamera. Active mode engages during standard surgical procedures,alignment mode during alignment of the master and slave devices, andcamera mode for the manipulation of the camera near the surgical site.

Master/slave surgical systems include one or more robotic surgical toolsconfigured to be manipulated by encoded signals. The encoded signalswill typically be electronic signals sent via wire or wirelessly from aremote location, where the operator is controlling the master device, tothe location where the surgical effectors are being roboticallymanipulated by the slave device.

The master device includes one or more surgical tool emulators whichcorrespond to at least some of the surgical effectors being controlledby the slave device at the surgical site. Oftentimes these emulators aremounted to and supported by the master device as a set of joysticks orsimilar.

In the embodiments presented hereafter, motion tracking technologyremoves the need for the joysticks used in traditional master/slavesystems. The master device comprises a motion tracking system to trackand interpret the motion of the emulators and computational system fortranslation of master device movements to slave device surgicaleffectors at the surgical site.

II. Motion Tracking System

FIG. 2 demonstrates a three-dimensional motion tracking system 200including a motion tracking apparatus 202 and a computational system204. The motion tracking apparatus is in the form of an open spacesurrounded by a system of detectors 206 mounted on supportive stanchions208. The system of detectors is capable of tracking the position andmotion of objects within a FWS 210. The system of detectors is coupledto the computational system such that encoded signals from the detectorscan be received and interpreted by the computational system which maygenerate a set of motion instructions for the slave device.

The motion tracking apparatus may be constructed as a controlledenvironment with the system of detectors at predefined locations withinthe environment and the FWS being confined within the controlledenvironment. The controlled environment may be configured as a box withthe one face removed, as a box with three adjacent faces and the otherfaces removed, as a box with two opposing faces and the other facesremoved, a platform, or the like.

The system of detectors may include a set of video cameras mounted tostanchions surrounding the FWS. The cameras are electronically coupledto a computational system capable of simultaneously inputting multiplevideo sources, recognizing independent objects within the FWS,calculating orientation metrics of objects within the FWS, interpolatingmultiple two-dimensional sets of orientation metrics into threedimensions, calculating the relative orientation metrics between objectswithin the FWS, and translating the orientation metrics as a set ofmotion instructions for the surgical tool.

The orientation metrics are independently monitored for each objectwithin the FWS (further elaborated below) and may include position,pitch, yaw, roll, speed, acceleration, distance, and similar.Alternatively, the system of detectors may include any method capable ofdetermining the orientation metrics of objects within the FWS includinginfrared sensors, acoustic sensors, and similar.

The FWS represents the area that objects and their orientation metricsare able to be accurately measured by the motion tracking apparatus. Theposition, size, and shape of the FWS may be determined by the relativepositions of the detectors making up the system of detectors.Additionally, the position, size, and shape of the FWS are limited bythe computational system to prevent movement of the surgical tooloutside a set of positional boundaries representing areas the surgicaltool should not (e.g. vulnerable tissue) or cannot access (e.g. outsideof slave motion constraints). Finally, the position size and shape ofthe FWS may be specifically constructed to represent the space beingmanipulated by the surgical system.

III. Operative Introduction to the Operative Modes of the Master/SlaveSurgical System

As briefly introduced above, the master/slave surgical system operatesin one of three modes: active, alignment, or camera. Active mode for useduring standard surgical procedures, alignment mode during alignment ofthe master and slave devices, and camera mode for the manipulation ofthe camera near the surgical site. FIGS. 3A-3E illustrates theseoperative modes of the master slave surgical system 300 and how changesin the orientation metrics of the emulators within the FWS translates tomovement of the effectors or camera at the surgical site in the variousoperative modes.

FIG. 3A illustrates a two-dimensional projection of the FWS 302 and theobjects within. The user's hands 304 and the emulators 306, hereafter incombination referred to as the emulators for clarity of description,within the FWS are monitored by the detector system configured to tracktheir motion 308 (i.e., orientation metrics) as part of the masterdevice in active mode.

FIG. 3B illustrates a video display 310 showing the slave surgicaleffectors 312 at the surgical site. When the master/slave surgicalsystem is operating in active mode 314, changes in the orientationmetrics of the emulators within the FWS are translated into motion ofthe surgical effectors 316 at the surgical site which is shown on thevideo display.

The video display component may be any number of devices capable ofrepresenting visual information such as a television screen, a computerscreen, projected images on a surface, a personal media display device,a heads up display or the like. The visual display may be generated by athree-dimensional camera, a stereoscopically-arranged pair of cameras,or a three-dimensional representation of space created from a pluralityof two-dimensional image sources, and similar. The corresponding visualdisplay may be viewed in a method used to present three-dimensionalvideo displays such as a stereoscopic device, goggles with augmentedreality, a virtual reality system, a holographic display system andsimilar.

FIG. 3C illustrates a similar system as FIG. 3A for tracking theemulators within a FWS using a detector system configured to track theirmotion 320 as part of the master device in camera mode.

FIG. 3D illustrates a video display showing the slave surgical effectorsat the surgical site while the master/slave system is operating incamera mode 330. While in camera mode, changes in the orientationmetrics of the emulators within the FWS are translated into motion ofthe camera at the surgical site. Motion of the camera at the surgicalsite moves the field of view 332 shown on the video display.

FIG. 3E illustrates a similar system as FIG. 3A for tracking emulatorswithin a FWS using a detector system configured to track their motion340 as part of the master device in alignment mode.

FIG. 3F illustrates a video display showing the slave surgical effectorsat the surgical site while the master/slave system is operating inalignment mode 350. While in alignment mode, motion of the effectors andthe camera at the surgical site are terminated, i.e. changes in theorientation metrics of the emulators within the FWS are not translatedinto motion of the camera or the effectors at the surgical site.

During operation of the master/slave surgical system, it may benecessary for the operator to change between operative modes. Thesechanges may be accomplished via a command to the master device. Thecommand may include a keyboard input, a mouse button click, a vocalcommand or similar. Alternatively, the master system may be calibratedto automatically shift between operative modes in specific circumstancesor situations that may be present at the slave device or the masterdevice. Several of these systems are discussed below.

Note that FIGS. 3A, 3C, and 3E are representative of a video inputreceived by the computational system from the detector system and do notnecessarily illustrate a video display showing the emulators within theFWS, however some systems may include such a display.

IV. Haptic Response to Indicate Mater-Slave Alignment has been Achieved.

During complex surgical procedures involving a master/slave surgicalsystem the master device translates the operator's actions into actionsperformed by the slave device at the surgical site. To do this, theoperator aligns the master and slave devices before the surgicalprocedure begins. During robotic surgeries the operator is inundatedwith the auditory and visual stimuli present in the operating room.Aligning the master and slave components without an auditory or visualcue may alleviate the stimulus saturation present in surgical situationsand facilitate easier operation. As one way to do this, FIGS. 4B-4C andthe discussion below illustrate a system for introducing haptic feedbackto assist the operator in the alignment of a master/slave surgicalsystem 400.

Similarly to FIG. 3A, FIG. 4A illustrates a two-dimensional projectionof the emulators within the FWS which are monitored by the detectorsystem configured to track their motion as part of the master device.The computational system concurrently monitors orientation metrics ofthe slave surgical effectors at the surgical site and the orientationmetrics of the emulators within the FWS. Additionally, the computationalsystem creates and monitors a set of orientation metrics within the FWSrepresenting the orientation metrics of the slave surgical effectors atthe surgical site, hereafter the representative effectors. Note that therepresentative effectors exist only within the computational system andare only illustrated in the Figures for ease of understanding.

Hereafter, current orientation metrics of an object monitored by thecomputational system are referred to as a position, e.g. the position ofthe emulators 402, and the position of the representative effectors 404;changes in orientation metrics of an object monitored by thecomputational system are referred to as movement, e.g. move (or movementof) the surgical effectors; and differences in the orientation metricsbetween separate objects monitored by the computational system arereferred to as a relative misalignment, e.g. the relative misalignmentbetween the representative effectors and the emulators.

Ideally, during active mode operation, there is no relative misalignmentbetween the representative effectors and the emulators. However, beforeor during surgical procedures a relative misalignment between therepresentative effectors and the emulators may be present or introducedover time due to various causes. When the relative misalignment betweenthe representative effectors and the emulators is above a certainthreshold quantitative or qualitative value, the system will enteralignment mode.

As illustrated in FIG. 4B, while the system is in alignment mode theuser may move the position of the emulators towards the position of therepresentative effectors in the FWS. When the relative misalignmentbetween the emulators and the representative effectors in the FWS isbelow a threshold defined in the computational system the emulator givesa haptic feedback response 406. The feedback response may be in the formof a pulse, vibration, click, temperature change, texture change, sizechange, or similar. Once the haptic feedback response has been given,the user may choose to leave alignment mode and enter active or cameramode.

Alternatively, rather than originating from the emulator, the hapticfeedback response may originate from a component associated with thesurgical tool system that is not the emulator, such as an armrest, awrist strap, eyewear, supportive surfaces, gloves, or similar. Further,there may be different forms of haptic feedback that each representdifferent aspects of the surgical procedure, such as relativemisalignment of the emulators and the surgical effectors, mishandling ofthe emulators, elapsing time, patient condition, and similar.

V. Handling Master Motion Outside Slave Motion Limits.

During a procedure utilizing a master/slave system to performrobotically assisted surgery, the slave device may have a limited rangeof motion due to the type of surgery, condition of the patient, nearbytissues types or various other limiting factors. When the slave deviceis operating in this type of environment, the user of the master devicemay move the emulators in the FWS to a position outside the range ofmotion of the surgical effectors and may create conditions that mayresult in a misalignment between the emulators and the effectors. Theembodiments described hereafter allow for the alignment of themaster/slave system after a relative misalignment between therepresentative effectors and the emulators near the limits of surgicaleffector movement 500.

A. Optimal Method for General Surgical Procedures.

FIG. 5A illustrates a two-dimensional projection of the position of theemulators and the position of the representative effectors within theFWS. The emulators are monitored by a detector system configured totrack their motion as part of the master device. The position of theemulators, the position of the representative effectors, and therelative misalignment between the two are all monitored by thecomputational system. As discussed above, the surgical effectors mayhave a limited range of motion and operate in a limited working area(LWA) 510. The computational system monitors the boundaries of the LWAand creates and monitors a corresponding set of representativeboundaries within the FWS 502, hereafter the representative LWA 502.

FIG. 5B illustrates a video display showing the surgical effectors atthe surgical site during active mode operation. The surgical effectorsoperate within boundaries of the LWA 510 and motion outside of thoseboundaries is not allowed. The LWA is monitored by the computationalsystem which also creates and monitors the representative LWA 502 withinthe FWS as shown in FIG. 5A. While there is no relative misalignmentbetween the emulators and the representative effectors and the surgicaleffectors remain within the LWA the master/slave surgical systemoperates in active mode.

FIG. 5C illustrates an instance in which the position of the emulatorsmoves 520 outside the boundaries of the representative LWA 502. Thesurgical effectors reach the boundaries of LWA at the surgical site(they are now at the limit of their range of motion) and cease movementcausing the position of the representative effectors to persist at theboundaries of the representative LWA 522. This creates a relativemisalignment between the position of the surgical effectors 530 and theposition of the emulators 524. When the relative misalignment measuredby the computational system moves above a threshold, the master/slavesurgical system may automatically enter alignment mode.

FIG. 5D illustrates the video display in the instance where the positionof the emulators moves outside the boundaries of the representative LWA502. The position of the surgical effectors at the surgical site reachthe limit to their range of motion and the boundaries of the LWA 530causing the surgical effectors cease movement.

While in alignment mode, the user works to reestablish alignment of themaster/slave surgical system. As shown in FIG. 5E, the user may move theposition of the emulators 540 towards the position of the representativeeffectors in the FWS. The position of the representative effectors haspersisted at the boundary of the representative LWA 542 because motionof the surgical effectors at the surgical site has been disallowed. Whenthe position of the emulators matches the position of the representativeeffectors within the FWS and the relative misalignment between the two544 drops below a threshold defined in the computational system, themaster/slave surgical system may automatically reenter active mode.

In some cases when the master/slave surgical system reenters activemode, the position of the emulators and the position of therepresentative effectors are in close proximity to boundaries of therepresentative LWA. Small, subsequent motions of the emulators mayquickly result in relative misalignments between the representativeeffectors and the emulators that result in automatically reenteringalignment mode.

To prevent this, when the master/slave surgical system initiallyreenters active mode, the computational system may create a new adjustedrepresentative LWA 546 which replaces the original representative LWA.The adjusted representative LWA still corresponds to the boundaries ofthe LWA at the surgical site; however, the boundaries of the adjustedrepresentative LWA have been moved within the FWS by the computationalsystem to allow more movement by the emulators before reaching theboundaries of the adjusted representative LWA and creating misalignmentthat would create a need to re-enter into alignment mode.

FIG. 5F illustrates the video display in the case where the position ofthe emulators and the position of the representative effectors match andthere is no longer relative misalignment. The position of the surgicaleffectors remains at the boundaries of the LWA 550 as the motion of theeffectors ceased when the master/slave surgical system automaticallyentered alignment mode. Now that the relative misalignment has beencorrected the tool may reenter active mode and movement of the surgicaleffectors again reflects motion of the emulators in the FWS, assumingthe emulator's motion remains within the adjusted representative LWA546.

The realignment system outlined in the above discussion of FIGS. 5A-5Ffor a master/slave surgical system utilizing gesture tracking technologyrepresents an exemplary embodiment. However, there are several otherembodiments to handle potential relative misalignment between theemulators and the representative effectors when the surgical effectorsare operating near the boundaries of the LWA.

B. Alternative Method Allowing Relative Misalignments Below a Tolerance.

During some surgical procedures, automatically entering alignment modedue to small relative misalignments when operating near the boundariesof the LWA may not be necessary. In this situation, the computationalsystem may have a secondary spatial or temporal tolerances associatedwith the relative misalignment threshold that triggers enteringalignment mode. The spatial tolerance allows slight movements above therelative misalignment threshold before entering alignment mode. Thetemporal tolerance allows movements with relative misalignment above thethreshold for a small period of time before entering alignment mode. Ifthe relative misalignment between the emulators and the representativeeffectors does not decrease below the threshold before the temporal orspatial tolerances are surpassed, the system will automatically enteralignment mode. If the relative misalignment decreases below thethreshold before the tolerances are surpassed, the system will remain inactive mode.

This would be represented on the video display as the surgical effectorstemporarily ceasing movement at the edge of the LWA without enteringalignment mode. The surgical effectors begin moving again if therelative misalignment is corrected. If the relative misalignment is notcorrected, the effectors will remain motionless and the master/slavesurgical tool enters alignment mode.

C. Alternative Method Allowing Relative Misalignments in Specific Areas.

Generally, the LWA (and the corresponding representative LWA) is athree-dimensional space with a set of boundaries in which the surgicaleffectors are able to move and operate. In many surgical procedures theboundaries of the LWA represent areas that should not be accessed by thesurgical effectors (e.g. vulnerable tissues) and motion of the emulatorsthat would cause movement of the effectors into these areas shouldtrigger cessation of effector movement.

However, in some surgical procedures, specific boundaries of the LWA mayrepresent areas that are non-critical. When the emulators move outsidethe boundary of the representative LWA in a non-critical area,automatically entering alignment mode may be detrimental to the overallsurgical procedure. In this situation, the motion tracking system mayinstead translate the movement of the emulators outside therepresentative LWA into movement of the surgical effectors representingsome projection of the movement onto the boundary of the LWA. This wouldbe demonstrated on the video display as the surgical effectors slidingalong the range of their motion limits without entering alignment mode.

As an alternative in a similar surgical procedure, when the emulatorsmove outside the representative LWA in a non-critical area the surgicaleffectors cease movement at the boundaries of the LWA at the surgicalsite. The emulators may reenter the boundaries of the representative LWAat a different non-critical position from where they exited therepresentative LWA. When this occurs, the surgical effectors mayautomatically move to a position within the boundaries of the LWAcomputed from the new position of the emulators in the representativeLWA. The position of the representative effectors is automatically setto the current position of the emulators after the movement of thesurgical effectors. This would be demonstrated on the video display asthe surgical effectors ceasing movement at one point at the boundary ofthe LWA and subsequently moving to a new position within the LWA aftersome time without entering alignment mode.

Any process that functions to realign the master/slave surgical system,for example those described in this subsection, may allow for thecomputation and creation of an adjusted representative LWA similar tothe system illustrated in FIG. 5E.

VI. Detection of User's Grip on the Master Controllers.

The master/slave surgical system relies on the operator to controlemulators on the master device that represent and control surgicaleffectors on the slave device

FIG. 6 shows a surgical emulator designed to identify unintendedemulator movements 600. The surgical emulator can be held in theoperator's hand and is designed to be a representation of the surgicaleffector at the surgical site. The emulator has at least one or moretouch activated sensors 602 and/or light emitting/reflecting surfaces604. The emulator may be wirelessly coupled or connected via umbilicalto the master device. The master device and the emulator may bothproduce and receive signals for controlling the state of any sensor,light emitting surface, or light reflective surface that may make up theemulator.

The touch activated sensors may be any component or set of componentsused to detect the grip of the operator, such as capacitive pads,force-sensitive resistors, switches, pressure-activated switches,infrared detector and emitter pairs, or similar. The light emitting orreflective surfaces may be any component or set of components that wouldbe occluded by the normal grip of the operator such as, light emittingdiodes, fluorescent materials, lasers, reflective tape, metallicsurfaces, or similar.

In an exemplary embodiment, the master device is wirelessly coupled tothe emulator which consisting of capacitive pads in an orientation suchthat the operator's normal grip on the emulator will create and transmitan encoded signal to the master device indicating that the emulator isbeing held in the operator's hand in an operative manner. In the eventthat the operator's grip becomes abnormal and changes the interactionwith the capacitive pads, the emulator will create and transmit anencoded signal to the master device indicating the emulator is not beingheld in an operative manner. When this signal is received the masterdevice may disengage active mode, cease the movement of the surgicaleffectors at the surgical site, and enter alignment mode until normalhandling of the emulator is restored.

In another design, the emulator utilizes light emitting diodes in anorientation such that the operator's normal grip on the emulatoroccludes the light from being emitted. The motion tracking system isfurther configured to detect the appearance of the light emitted by thelight emitting diodes when the operator's normal grip is compromised.When the light is detected, the master device may disengage active mode,cease the movement of the surgical effectors at the surgical site, andenter alignment mode until normal handling of the emulator is restored.

Alternatively, the emulator may not be specifically designed to signifychanges in the users grip. The detector system and the computationalsystem may be further configured to recognize the operator's normal andabnormal grip on the emulator via object, gesture, or motion recognitionalgorithms.

VII. Camera Control of a Master/Slave System that Prevents RotationalMisalignment

During the operation of a master/slave surgical system in active mode,the orientation of the medical devices (e.g., surgical effectors, tools,devices, or instruments) may make viewing specific areas difficult whenthe medical devices are in the desired line of sight of the cameramonitoring a specific area at the target site (e.g., a surgical site).The ability to manipulate the position of camera at the surgical siteand the corresponding field of view on the video display using theemulators is advantageous, providing a more streamlined surgicaloperation.

FIGS. 7A-7F depict a camera control system for a master/slave surgicalsystem 700. In one aspect of the disclosure, the system 700 comprisesemulator(s) (e.g., the emulators 402) configured to be held and operatedin a FWS (e.g., the FWS 302), the emulator(s) representing medicaldevice(s)(e.g. surgical effectors) at the target site. The system 700may comprise one or more detectors (e.g., the detectors 206 or imagingdevice(s)) may be configured to track the emulator(s) within the FWS.The system 700 may comprise; at least one computer-readable memoryhaving stored thereon executable instructions; and at least oneprocessor in communication with the at least one computer-readablememory and configured to execute the instructions to cause the system700 to perform steps as described below. In camera mode, the motiontracking system interprets translational movement of the emulators to betranslational movement of the camera position. Consequently, in cameramode translational movement of the emulator does not correspond/resultin translational movement of the surgical effectors. However, in cameramode rotational motion of the emulators does result in rotational motionof the surgical effectors. Consequently, in camera mode rotationalmovement of the emulator does not result in rotational motion of thecamera.

In this example, FIG. 7A illustrates a two-dimensional projection of theposition of the emulators and the position of the representativeeffectors within the FWS. While in active mode, the translationalmovement 702 of the emulators is monitored by a detector systemconfigured to track the emulator motion as part of the master device.

FIG. 7B illustrates a video display 310 showing the surgical effectorsat the surgical site during active mode operation. The movement of theemulators is translated into motion of the surgical effectors at thesurgical site 710. While in active mode, translational movement 712 androtational movement 714 of the camera are not allowed.

To continue, as illustrated in FIG. 7C, the operator may engage cameramode to change the position of the camera and the corresponding field ofview on the video display. In this mode, translational movement of theemulators 720 within the FWS is monitored by the detector system.

FIG. 7D illustrates that in camera mode, translational movement of thecamera is allowed and a translational movement of the emulators does notgenerate movement for the surgical effectors 730, but instead generatestranslational movement of the camera 732 altering the field of view onthe video display.

Finally, as illustrated in FIG. 7E, rotational movement of the emulators740 within the FWS is monitored by the detector system while still incamera mode.

FIG. 7E illustrates that in camera mode a rotational movement of theemulators generates movement for the surgical effectors limited torotational motions 750. Rotational movement of the camera is disallowedin camera mode.

Note that FIGS. 7A, 7C, and 7E are representative of a video inputreceived by the computational system from the detector system and do notillustrate a video display showing the emulators within the FWS.

In related aspects, the at least one processor may be configured toexecute the instructions to cause the system to: receive a signal fromat least one detector indicative of a translational movement of theemulator within the FWS; and generate instructions, based on thetranslational movement of the emulator, to move an imaging device withina plane defined by pitch and yaw axes of the imaging device. In oneembodiment, translational movement of the emulator does not result in atranslational movement of the medical device.

The at least one processor may be further configured to execute theinstructions to cause the system to receive a signal from the at leastone detector indicative of a rotational movement of the emulator withinthe FWS; and generate instructions, based on the rotational movement ofthe emulator, to rotate the medical device along a roll axis of themedical device. In one embodiment, rotational movement of the emulatordoes not result in a rotational movement of the imaging device.

VIII. Automatic Ratcheted Elimination of Master Slave Alignment Offsets

During operation of the master/slave surgical system in active mode, therelative alignment offset (i.e., misalignment) between the emulators andthe representative medical devices (e.g., surgical effectors, tools,devices, or instruments) in the FWS may increase but remain below thetolerance that would trigger a transition from active mode to alignmentmode. The slight misalignment is allowed to reduce the total time ittakes for the user to achieve adequate alignment.

However, it is desirable to reduce misalignment without necessarilyrequiring activation of alignment mode or any other additional effortfrom the user.

In one aspect of the disclosure, a system may comprise: an emulatorconfigured to be held and operated in a FWS, the emulator representing amedical device at a target site; at least one detector configured totrack the emulator within the FWS; at least one computer-readable memoryhaving stored thereon executable instructions; and at least oneprocessor in communication with the at least one computer-readablememory and configured to execute the instructions to cause the system toperform the method as described below.

FIGS. 8A-8F depict a representation of a method to automatically reducethe relative misalignment between the emulators and the representativeeffectors within the FWS 800 using what is herein referred to as“ratcheting” technique, which is described in the following paragraphs.

FIG. 8A depicts a linear representation of the position of the emulators802 and a linear representation of the position of the representativeeffectors 804. Additionally. FIG. 8B depicts a linear representation ofthe position of the surgical effectors 810.

In this example, λ_(n) is the relative misalignment between the positionof the emulators and the position of the representative surgicaleffectors. α_(n) is a movement of the emulators the generates acorresponding movement of the effectors ω_(n), where the two willgenerally differ from each other with respect to position orientationmetrics but not rotation orientation metrics. φ_(n) is an additionalmovement of the emulators to create a total movement θ_(n) that reducesthe relative misalignment λ_(n).

To elaborate using FIGS. 8A and 8B, the surgical system is operating inactive mode with a relative misalignment λ₁ between the emulators andthe representative effectors. The operator moves the emulators by α₁ tocreate a corresponding movement of the slave surgical effectors ω₁ atthe target site, such as, for example, the surgical site. Traditionally,an operator movement of α₁ creates a direct translation of motion to thesurgical effectors ω₁. In this example, the computational systemcomputes an additional (or lesser) amount of movement φ₁ as a functionof λ₁, ω₁, α₁ that when added to (or subtracted from) ai represents thetotal actual movement of the emulators θ₁ (i.e., θ₁=α₁+φ₁=ω₁+ƒ (ƒ₁, ω₁,α₁)) that creates the movement ω₁. The movement of the emulators by θ₁yields a lesser relative misalignment between the emulators and therepresentative effectors.

This example continues using FIGS. 8C and 8D wherein the same processoccurs with λ₂ being the lesser relative misalignment between theemulators and the representative effectors resulting from the totalmovement θ₁. The computational system computes an additional (or lesser)amount of movement φ₂ as a function of λ₂, ω₂, α₂ that when added to (orsubtracted from) α₂ represents the total actual movement of theemulators θ₁. The movement of the emulators by θ₂ yields an even smallerin magnitude relative misalignment between the emulators and therepresentative effectors. This process continually iterates as in FIGS.8E and 8F until the relative misalignment λ_(n) is neutralized.

It is possible for the computational system to calculate a specificadditional movement φ′_(n) that would neutralize the relativemisalignment in a single iteration. While efficient, the singleiteration may create a movement by the user that is unnatural (e.g.extreme over motions to compensate for smaller misalignments) anddetrimental to the overall medical or surgical procedure. The totalamount of movement θ′_(n) to neutralize the relative misalignment isbuilt in a series of smaller additive movements over time, i.e.θ′_(n)=Σ_(i) ^(j)φ_(i) where j is the number of iterations required toneutralize the relative misalignment. This iterative process can bedescribed as ‘ratcheting’ down the misalignment, and the larger thenumber of iterations j the smoother the ratcheting process appears tothe operator.

In related aspects, the at least processor may be configured to executethe instructions to cause the system to: determine an alignment offsetbetween a location of the emulator and a location of the medical device;during a medical procedure, determine a first movement amount based on asignal from the at least one detector indicative of a first movement ofthe emulator within the FWS; adjust the first movement amount by a firstadjustment value; and generate instructions to move the medical devicebased on the adjusted first movement amount, wherein movement of themedical device by the adjusted first movement amount reduces thealignment offset between the location of the emulator and the locationof the medical device.

The at least one processor may be further configured to execute theinstructions to cause the system to repeat the steps described above tofurther reduce the alignment offset. For example, the at least oneprocessor may be further configured to execute the instructions to causethe system to: during the medical procedure, determine a second movementamount based on a signal from the at least one detector indicative of asecond movement of the emulator within the FWS; adjust the secondmovement amount by a second adjustment value; and generate instructionsto move the medical device based on the adjusted second movement amount,wherein movement of the medical device by the adjusted second movementamount reduces the alignment offset between the location of the emulatorand the location of the medical device. In one embodiment, the movementof the medical device by the adjusted first and second movement amountsmay eliminate the alignment offset between the location of the emulatorand the location of the medical device. In another embodiment, thealignment offset between the location of the emulator and the locationof the medical device may be eliminated after more than two adjustmentsor movements of the medical device.

IX. Tracking System that Self-Monitors the Quality of the Calibration.

The master/slave surgical system controlled via an optical motiontracking system requires calibration with one or more known objects in aknown geometry before it can provide accurate tracking data. Thecalibration can be affected by many factors, such as lighting andtemperature, and may change or degrade over time. It is desirable tomaintain optimal calibration with minimal intervention from the user tocreate accurate translations of motion from the master device to theslave device in an efficient manner.

FIG. 9 represents a motion tracking system that self-monitors thequality of the calibration 900. The position of the emulators within theFWS are monitored by the detector system configured to track theirmotion, depicted by a two-dimensional projection of the FWS and theobjects within. The configuration of the detector system utilizes one ormore known objects in a known orientation 902 in the field of view ofthe detector system.

Alternatively, the known objects may be a set of light sources orreflective surfaces that can be monitored by the detector system. Thelight sources may be light emitting diodes, infrared emitters, lasers,or similar. The reflective surfaces may be reflective tapes, metals, orsimilar. The known objects may also be a set of objects or surfaces.

The operator calibrates the motion tracking system with an orientationbaseline using the known objects while the tool is in alignment mode.Once the operator engages active mode, the computational system activelycompares the position of objects within the FWS to the orientationbaseline created during calibration.

In the case where the relative misalignment between the measuredposition of the objects within the FWS and the orientation baseline isabove a tolerance threshold the tool may indicate to the user thatcalibration has been compromised. Alternatively, when the misalignmentbetween the measured position of the objects within the FWS and theorientation baseline is above a tolerance threshold the toolautomatically recalibrates the orientation baseline using the positionof the known objects.

X. View Based Automatic Adjustment of Master/Slave Ratio.

The surgical system has a translation ratio that represents theproportion of the magnitude of motion (change in orientation metric)that is translated from the emulators to the effectors. Depending onthis ratio, an operator may have to move the emulators a greater orlesser distance to cause a certain amount of motion in the effectors.

If a surgical system has only a fixed translation ratio, or atranslation ratio that is not easily changeable during surgicalprocedures, it is difficult for the operator to rapidly change or movebetween magnifications or fields of view used on the video display.

To address this, the master/slave surgical system is capable of motionscaling. For example, surgical effectors of the slave device may requiremuch less movement than the operator's hands use in moving theemulators, allowing the operator to make comparatively larger motions tocause comparatively smaller/finer actions of the surgical effectors.

FIG. 10 represents a system that allow a high level of surgical controlat various video display magnifications without requiring the surgeon toissue separate commands to adjust the translation ratio between themaster device and slave device 1000.

As illustrated in FIG. 10A, the operator of the master device uses avideo display comprising a wearable set of stereoscopic goggles 1002that display an image of the surgical effectors at the surgical site.The goggles consist of a system that monitors and measures the distancebetween the goggle eyepieces and the emulators, hereafter themagnification distance 1004. The movement of the emulators 1006 isconverted by the translation ratio 1008 into a movement of the surgicaleffectors 1010 based on the magnification distance. Thus, a change in amagnification distance, for example by the operator moving the googlescloser to or further from the emulators, creates a corresponding changein the magnification distance, and therefore in the translation ratio.

A specific example of this is demonstrated in FIG. 10B. At a smallermagnification distance 1012 the translation ratio creates a smallermovement 1014 in the surgical effectors for a similar movement of theemulators. Similarly, though not shown, at a larger magnificationdistance the translation ratio creates a larger movement of the surgicaleffectors for a similar movement of the emulators.

The system that monitors and measures the magnification distance maycomprise any components or set of components that would allow for themeasurement of the magnification distance such as stereoscopic camerasmounted on the goggles, light sources and detectors mounted on thegoggles and emulators, a camera mounted on the goggles and calibrationmarkers on the emulator, or similar.

Alternatively, the magnification distance may be monitored by the samemotion tracking system that monitors the position of the emulators, orby a motion tracking system configured to track the position of theoperator's head or eyes relative to the emulators.

FIG. 11 represents another system which enables a high level of surgicalcontrol at various video display magnifications without forcing thesurgeon to issue separate commands to adjust the translation ratiobetween the master device and slave device 1100.

As illustrated in FIG. 11A, the operator of the master device uses avideo display 1102 with a scalable magnification setting that displaysthe surgical effectors at the surgical site. At any point in time, thevideo display has a specific magnification setting 1104. A movementchange of the emulators 1106 is converted by the translation ratio 1108into movement of the surgical effectors 1110 based on the specificmagnification setting.

A specific example of this is demonstrated in FIG. 11B. At a largermagnification distance 1112 the translation ratio creates a smallermovement 1114 in the surgical effectors for a similar movement of theemulators. Similarly, though not shown, at a larger magnificationdistance the translation ratio creates a larger movement of the surgicaleffectors for a similar movement of the emulators.

XI. Clutched Roll

In one aspect of the present disclosure, there is provided a systemwherein an operator (e.g., a physician or surgeon) controls an emulatorthat represents and controls a medical device (e.g., surgical effectors,tools, devices, or instruments). The system may be configured to controlan instrument with extended roll capability, such as, for example, amedical device with an end-effector controlling or otherwisemanipulating a curved needle.

The ability to control the medical device may be limited by the rollcapability of an operator's wrist. For example, the operator may not beable to rotate his/her wrist beyond certain angles—for example, about66.1° for pronation (i.e., rotation of the wrist such that the palm isfacing downward or backward/posteriorly) or about 75° for supination(i.e., rotation of the wrist such that the palm is facing upward orforward/anteriorly). Thus, it may be desirable for the system to enablethe operator to employ the extended roll capability of the medicaldevice (e.g., extending the roll capability beyond that of theoperator's wrist) and not be limited or hampered by the anatomicallimitation of the operator (e.g., anatomical limitation of theoperator's wrist). This can help the operator conduct tasks requiringextensive instrument roll, such as, for example, during suturing.

One aspect of the disclosure provides the operator with the ability toperform clutched roll of the medical device, allowing the operator toactivate a clutch (e.g., a clutch foot pedal, button, or other input toactivate the decoupling of the absolute roll angle of the medical devicefrom that of the emulator) and thereby roll the medical device fartherthan his/her wrist could go. This clutched roll feature allows theoperator to use a ratcheting turning motion to roll the instrument inone motion, pause the roll, and then continue rolling the instrument ina subsequent motion, while, in some cases, maintaining wrist yaw-pitch.

The clutched roll feature of the system facilitates decoupling anabsolute roll angle of the medical device from an absolute roll angle ofthe emulator. The clutched roll of the medical device allows fordecoupling the roll axis from the other axes of the medical device(i.e., the yaw axis or the pitch axis). In other words, rotationalmovements with respect to the roll axis of the emulator are nottranslated into rotational movements with respect to the roll axis ofthe medical device, whereas the rotational movements of the emulatorwith respect to the other axes are translated to the medical device.

FIGS. 12A-12F illustrate a system 1200 configured to perform a clutchedroll. The system 1200 may comprise an emulator 1206 configured to beheld and operated in a FWS (e.g., the FWS 302), the emulatorrepresenting a medical device 1214 at a target site; one or moredetectors (e.g., detectors 206) configured to track the emulator 1206within the FWS 302; one or more detectors configured to track themedical device 1214; at least one computer-readable memory having storedthereon executable instructions; and at least one processor incommunication with the at least one computer-readable memory andconfigured to execute the instructions to cause the system to perform aclutched roll as described below.

FIG. 12A illustrates a two-dimensional projection of the position of theemulator 1206 and the position of the representative medical device 1214within the FWS 302. The movement of the emulator 1206 (e.g., clockwiserotation by 90 degrees) is monitored by a detector system configured totrack the motion of the emulator.

FIG. 12B illustrates a video display 310 showing the medical device atthe target site during the non-clutched mode operation. During thenon-clutched mode, the rotational movement 1208 of the emulator 1206(e.g., clockwise rotation by 90 degrees) in all axes is translated intoa rotational motion 1216 of the medical device 1214 at the target site(e.g., clockwise rotation by 90 degrees). In one aspect, the system 1200may rotate the medical device 1214 based on the rotational movement ofthe emulator 1206 by, for example, conducting a spherical linearinterpolation (slerp) of the current medical device quaternion towardsthe emulator quaternion. During the non-clutched mode, the operator mayrotate the emulator with his/her wrist with respect to its roll axis asfar as the anatomical structure of the wrist allows.

As illustrated in FIG. 12C, when the operator reaches the limit of howfar he/she can rotate his/her wrist, the operator may activate aclutched roll mode of the system to decouple an absolute roll angle ofthe medical device 1214 from an absolute roll angle of the emulator1206. As shown in FIG. 12D, in the clutched roll mode (as signified by amark or icon 1230 on the video display 310), a rotational movement 1210of the emulator 1206 (e.g., counterclockwise rotation by 90 degrees)within the FWS 302 with respect to its roll axis is not translated intoa rotational movement 1218 of the medical device 1214 with respect toits roll axis. As a result, during the clutched roll mode, the operatormay rotate his/her wrist to its original position with respect to itsroll axis without moving the medical device 1214.

In one aspect of the disclosure, steps to conduct the clutched roll (asmay be performed by the system 1200) are described below as follows.First, a cross product and a dot product of the z-vector of the emulator1206 (e.g., a unit vector along the roll axis of the master device) andthe z-vector of the medical device 1214 (e.g., a unit vector along theroll axis of the slave device) are computed. Then, the instantaneousrotation matrix is calculated using the following matrix formula:

$R = \begin{bmatrix}{\frac{{{cp}\lbrack 0\rbrack} \cdot {{cp}\lbrack 0\rbrack}}{wa} + {ca}} & {\frac{{{cp}\lbrack 0\rbrack} \cdot {{cp}\lbrack 1\rbrack}}{wa} - {{cp}\lbrack 2\rbrack}} & {\frac{{{cp}\lbrack 0\rbrack} \cdot {{cp}\lbrack 2\rbrack}}{wa} + {{cp}\lbrack 1\rbrack}} \\{\frac{{{cp}\lbrack 1\rbrack} \cdot {{cp}\lbrack 0\rbrack}}{wa} + {{cp}\lbrack 2\rbrack}} & {\frac{{{cp}\lbrack 1\rbrack} \cdot {{cp}\lbrack 1\rbrack}}{wa} + {ca}} & {\frac{{{cp}\lbrack 1\rbrack} \cdot {{cp}\lbrack 2\rbrack}}{wa} - {{cp}\lbrack 0\rbrack}} \\{\frac{{{cp}\lbrack 2\rbrack} \cdot {{cp}\lbrack 0\rbrack}}{wa} - {{cp}\lbrack 1\rbrack}} & {\frac{{{cp}\lbrack 2\rbrack} \cdot {{cp}\lbrack 1\rbrack}}{wa} + {{cp}\lbrack 0\rbrack}} & {\frac{{{cp}\lbrack 2\rbrack} \cdot {{cp}\lbrack 2\rbrack}}{wa} + {ca}}\end{bmatrix}$

wherein

R=instantaneous rotation matrix

cp=a cross product vector of the z-vector of the emulator 1206 and thez-vector of the medical device 1214:

ca=a scalar of a cross product of the z-vector of the emulator 1206 andthe z-vector of the medical device 1214; and

wa=a dot product of the z-vector of the emulator 1206 and the z-vectorof the medical device 1214.

The instantaneous rotation matrix R allows the alignment of the rollaxis of the emulator 1206 and the roll axis of the medical device 1214without consideration of the other axes (i.e., pitch and yaw axes) ofthe emulator 1206 or the medical device 1214. Then, the targetorientation of the medical device 1214 is calculated from the currentorientation of the slave device and the instantaneous rotation matrix asfollows:

v_(f) = R × v_(i)

wherein v_(i),=current orientation of the medical device 1214:

v_(f)=target orientation of the medical device 1214; and

R=instantaneous rotation matrix.

Then, the medical device 1214 is rotated based on the target orientationcalculated above. The rotation of the medical device 1214 may beconducted by a variety of techniques including, but not limited to slerpor linear interpolation (lerp).

Finally, as illustrated in FIG. 12E, the operator can turn off theclutched roll mode and repeat rotating the emulator 1206. As shown inFIG. 12F, during the non-clutched mode, the rotational movement 1212 ofthe emulator 1206 (e.g., clockwise rotation by 90 degrees) in all axesis translated into a rotational motion 1220 of the medical device 1214at the target site (e.g., clockwise rotation by 90 degrees). Theoperator may repeat the first step and the second step until the desiredrotation is achieved.

Note that FIGS. 12A, 12C, and 12E are representative of a video inputreceived by the computational system from the detector (e.g., detectors206) and do not illustrate a video display showing the emulators withinthe FWS. In actual use of one embodiment, the operator's hand would beholding the emulator 1206. However, for purposes of showing theorientation of the emulator 1206 and the operator's hand in the FWS, theoperator's hand is shown next to the emulator 1206 rather than over orcovering the emulator 1206.

In one aspect of the disclosure, the surgical system may be monitoredand/or controlled via a motion tracking system. In other words, one ormore locations or motions of the medical device and/or the emulator maybe tracked by the motion tracking system. The motion tracking system maybe optical or electromagnetism (EM)-based. In another aspect of thedisclosure, the clutched roll feature may be used in a system that doesnot utilize the motion tracking system. In one aspect of the disclosure,features disclosed in other sections may be used in a system that doesnot utilize motion tracking. In one aspect of the disclosure, theemulator may be a mechanical emulator, and the detectors may beconfigured to track the mechanical movement of the mechanical emulatorand/or the operation of the mechanical emulator by the operator.

In related aspects, the system may comprise an emulator representing amedical device at a target site. The system may comprise a first set ofone or more detectors configured to track the emulator. The system maycomprise a second set of one or more detectors configured to track themedical device at the target site. The system may comprise at least onecomputer-readable memory having stored thereon executable instructions;and at least one processor in communication with the at least onecomputer-readable memory and configured to execute the instructions tocause the system to: receive, from the first set of one or moredetectors, first data indicative of at least an orientation of theemulator, the first data comprising roll data, pitch data, and yaw dataof the emulator; generate, based on a clutched user input, instructionsto move the medical device based on the first data discounting the rolldata of the emulator; and cause the medical device to move based on theinstructions. In one embodiment, the emulator may be configured to beheld and operated in a FWS; and the first set of one or more detectorsmay be configured to track motion of the emulator in the FWS. In oneembodiment, the emulator may comprise a mechanical emulator; and thefirst set of one or more detectors may be configured to track mechanicalmovement of the emulator.

In one embodiment, the discounting of the roll data of the emulator maybe based on decoupling a roll axis of the emulator from yaw and pitchaxes of the emulator. In one embodiment, the discounting of the rolldata of the emulator may be based on decoupling an absolute roll angleof the medical device from an absolute roll angle of the emulator. Inone aspect, the movement of the medical device based on the instructionsmay facilitate adjustment of a roll axis of the emulator with respect toa roll axis of the medical device.

In one embodiment, the at least one processor may be configured toexecute the instructions to cause the system to receive, from the secondset of one or more detectors, second data indicative of an orientationof the medical device at the target site, the second data comprisingroll data, pitch data, and yaw data of the medical device; and thealignment of the respective roll axes of the emulator and the medicaldevice is based on the pitch and yaw data of the emulator and the pitchand yaw data of the medical device.

In one embodiment, the at least one processor may be configured toexecute the instructions to cause the system to: receive, from the firstset of one or more detectors, third data indicative of a translationalmovement of the emulator; receive, from the second set of one or moredetectors, fourth data indicative of a position of the medical device atthe target site; and generate instructions to move the medical devicebased on the third and fourth data.

In one aspect of the disclosure, the emulator may be axially symmetricor axially asymmetric. FIGS. 13A and 13B illustrate an exemplaryemulator 1300 that is axially symmetric. As shown in FIG. 13A, theemulator 1300 comprises a handle 1302, a connector 1304, and arms 1306and 1308. The operator may hold the emulator 1300 by either the handle1302 or the connector 1304 (as shown by the hand 1310 of the operator).Each of the arms 1306 and 1308 may comprise one or more branches 1307and 1309, respectively. The branches of the arm may be positionedsymmetrically with respect to the longitudinal axis of the arm (e.g.,branches 1309 of the arm 1308) or asymmetrically with respect to thelongitudinal axis of the arm (e.g., branches 1307 of the arm 1306). Thebranches 1307 and 1309 of each arm 1306 or 1308 may be positioned to beon the same plane. The plane formed by the branches 1307 and the planeformed by the branches 1309 may, for example, be perpendicular to eachother. As shown in FIG. 13B, the connector 1304 may comprise a pluralityof legs. The longitudinal length of the connector 1304 may be modifiedby adjusting the bending degree of bending of legs of the connector1304. The connector 1304 and the handle 1302 may be connected by a jointsuch that the connecting angle between the connector 1304 and the handle1302 may be adjusted based on different grip positions of the operator.The axially symmetric emulator 1300 may be beneficial because it canavoid an implication of mapping an operator grip roll angle (e.g., rollangle of the emulator 1300) to the instrument grip roll angle (e.g.,roll angle of the medical device). In some aspects, the emulator may besymmetric with respect to one or more of the pitch, yaw, and roll axes.

FIG. 14 illustrates an exemplary emulator 1400 that is axiallyasymmetric. The emulator 1400 comprises a handle 1402 and arms 1406 and1408. Each of the arms 1406 and 1408 may comprise one or more branches1407 and 1409, respectively. The handle 1402 of the emulator is shapedto be axially asymmetric. When the non-axially symmetric emulator 1400is used, an angular offset experienced by the operator (which is commonin traditional manual medical devices that allow the device shaft to berotated with respect to the handle) may be compensated for.

FIG. 15 illustrates an exemplary emulator 1500 that is axiallyasymmetric. The emulator 1500 comprises one or more rings 1502, one ormore pinchers 1504, a rod 1506, and a support 1508. The one or morerings 1502 are configured to receive fingers of the operator (as shownby the hand 1510 of the operator). Each of the rings 1502 is connectedto one of the pinchers 1504 and enables the operator to manipulate thepinchers 1504 when the operator inserts his/her fingers into the rings1502. One or more pinchers 1504 are connected to the rod 1506, and therod 1506 is mounted on the support 1508. Tips of the pinchers 1504 maybe pushed toward the longitudinal axis of the rod 1506, and the movementof the pinchers 1504 may be tracked by one or more detectors (e.g.,sensors inside the rod 1506). In some aspects, the emulator may beasymmetric with respect to one or more of the pitch, yaw, and roll axes.

In some aspects, the emulator 1500 may be used in conjunction with oneor more medical devices that are configured to pinch or clamp or operateby a pinching movement of the operator, including but not limited toforceps, clamps, scissors, and vessel sealers. When the operator pushesthe pinchers 1504 using his/her fingers toward the longitudinal axis ofthe rod 1506, the movement of the pinchers 1504 may be detected by thesensors, and the medical device may be operated or moved based on themovement of the pinchers 1504.

Implementing Systems and Terminology

Implementations disclosed herein provide systems, methods and apparatusfor to robotically controlling a medical device. More specifically,implementations of the present disclosure relate to a system forreducing an alignment offset; a camera control system; and a system fora clutched roll.

It should be noted that the terms “couple,” “coupling,” “coupled” orother variations of the word couple as used herein may indicate eitheran indirect connection or a direct connection. For example, if a firstcomponent is “coupled” to a second component, the first component may beeither indirectly connected to the second component via anothercomponent or directly connected to the second component.

The methods described herein may be stored as one or more instructionson a processor-readable or computer-readable medium. The term“computer-readable medium” refers to any available medium that can beaccessed by a computer or processor. By way of example, and notlimitation, such a medium may comprise random-access memory (RAM),read-only memory (ROM), electrically erasable programmable read-onlymemory (EEPROM), flash memory, compact disc read-only memory (CD-ROM) orother optical disk storage, magnetic disk storage or other magneticstorage devices, or any other medium that can be used to store desiredprogram code in the form of instructions or data structures and that canbe accessed by a computer. It should be noted that a computer-readablemedium may be tangible and non-transitory. As used herein, the term“code” may refer to software, instructions, code or data that is/areexecutable by a computing device or processor.

The methods disclosed herein comprise one or more steps or actions forachieving the described method. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isrequired for proper operation of the method that is being described, theorder and/or use of specific steps and/or actions may be modifiedwithout departing from the scope of the claims.

As used herein, the term “plurality” denotes two or more. For example, aplurality of components indicates two or more components. The term“determining” encompasses a wide variety of actions and, therefore,“determining” can include calculating, computing, processing, deriving,investigating, looking up (e.g., looking up in a table, a database oranother data structure), ascertaining and the like. Also, “determining”can include receiving (e.g., receiving information), accessing (e.g.,accessing data in a memory) and the like. Also, “determining” caninclude resolving, selecting, choosing, establishing and the like.

The phrase “based on” does not mean “based only on,” unless expresslyspecified otherwise. In other words, the phrase “based on” describesboth “based only on” and “based at least on.”

The previous description of the disclosed implementations is provided toenable any person skilled in the art to make or use the presentdisclosure. Various modifications to these implementations will bereadily apparent to those skilled in the art, and the generic principlesdefined herein may be applied to other implementations without departingfrom the scope of the present disclosure. For example, it will beappreciated that one of ordinary skill in the art will be able to employa number of corresponding alternative and equivalent structural details.Thus, the present disclosure is not intended to be limited to theimplementations shown herein but is to be accorded the widest scopeconsistent with the principles and novel features disclosed herein.

What is claimed is:
 1. A system, comprising: an emulator configured tobe held and operated in a free working space (FWS), the emulatorrepresenting a medical device at a target site; at least one detectorconfigured to track the emulator within the FWS; at least onecomputer-readable memory having stored thereon executable instructions;and at least one processor in communication with the at least onecomputer-readable memory and configured to execute the instructions tocause the system to: determine an alignment offset between a location ofthe emulator and a location of the medical device; during a medicalprocedure, determine a first movement amount based on a signal from theat least one detector indicative of a first movement of the emulatorwithin the FWS; adjust the first movement amount by a first adjustmentvalue; and generate instructions to move the medical device based on theadjusted first movement amount, wherein the movement of the medicaldevice by the adjusted first movement amount reduces the alignmentoffset between the location of the emulator and the location of themedical device.
 2. The system of claim 1, wherein the alignment offsetcomprises a plurality of adjustment values configured to iterativelyreduce the alignment offset, the plurality of adjustment valuesincluding at least the first adjustment value and a second adjustmentvalue.
 3. The system of claim 1, wherein the first adjustment value is afunction of at least the first movement of the emulator, the alignmentoffset between the location of the emulator and the location of themedical device, and the first movement amount.
 4. The system of claim 1,wherein the alignment offset between a location of the emulator and alocation of the medical device is expressed as θ′_(n)=Σ_(i) ^(j)φ_(i),wherein θ′_(n) is equal to the alignment offset, j is a number ofiterations to reduce the alignment offset between a location of theemulator and a location of the medical device to zero, and φ_(i) is thefirst adjustment value.
 5. The system of claim 1, wherein the at leastone processor is configured to execute the instructions to cause thesystem to: during the medical procedure, determine a second movementamount based on a signal from the at least one detector indicative of asecond movement of the emulator within the FWS; adjust the secondmovement amount by a second adjustment value; and generate instructionsto move the medical device based on the adjusted second movement amount,wherein the movement of the medical device by the adjusted secondmovement amount reduces the alignment offset between the location of theemulator and the location of the medical device.
 6. The system of claim5, wherein the movement of the medical device by the adjusted firstmovement amount and the adjusted second movement amount eliminates thealignment offset between the location of the emulator and the locationof the medical device.
 7. The system of claim 1, wherein the at leastone processor is configured to execute the instructions to cause thesystem to: receive, from the at least one detector, first dataindicative of at least an orientation of the emulator, the first datacomprising roll data, pitch data, and yaw data of the emulator.
 8. Amethod, comprising: at a system having an emulator and at least onedetector, wherein the emulator is configured to be held and operated ina free working space (FWS), the emulator representing a medical deviceat a target site, and the at least one detector is configured to trackthe emulator within the FWS: determining an alignment offset between alocation of the emulator and a location of the medical device; during amedical procedure, determining a first movement amount based on a signalfrom the at least one detector indicative of a first movement of theemulator within the FWS; adjusting the first movement amount by a firstadjustment value; and generating instructions to move the medical devicebased on the adjusted first movement amount, wherein the movement of themedical device by the adjusted first movement amount reduces thealignment offset between the location of the emulator and the locationof the medical device.
 9. The method of claim 8, wherein the alignmentoffset comprises a plurality of adjustment values configured toiteratively reduce the alignment offset, the plurality of adjustmentvalues including at least the first adjustment value and a secondadjustment value.
 10. The method of claim 8, wherein the firstadjustment value is a function of at least the first movement of theemulator, the alignment offset between the location of the emulator andthe location of the medical device, and the first movement amount. 11.The method of claim 8, wherein the alignment offset between a locationof the emulator and a location of the medical device is expressed asθ′_(n)=Σ_(i) ^(j)φ_(i), wherein θ′_(n) is equal to the alignment offset,j is a number of iterations to reduce the alignment offset between alocation of the emulator and a location of the medical device to zero,and φ_(i) is the first adjustment value.
 12. The method of claim 8,further comprising: during the medical procedure, determining a secondmovement amount based on a signal from the at least one detectorindicative of a second movement of the emulator within the FWS;adjusting the second movement amount by a second adjustment value; andgenerating instructions to move the medical device based on the adjustedsecond movement amount, wherein the movement of the medical device bythe adjusted second movement amount reduces the alignment offset betweenthe location of the emulator and the location of the medical device. 13.The method of claim 12, wherein the movement of the medical device bythe adjusted first movement amount and the adjusted second movementamount eliminates the alignment offset between the location of theemulator and the location of the medical device.
 14. The method of claim8, further comprising: receiving, from the at least one detector, firstdata indicative of at least an orientation of the emulator, the firstdata comprising roll data, pitch data, and yaw data of the emulator. 15.A non-transitory computer readable storage medium comprising one or moreprograms, the one or more programs comprising computer-executableinstructions, which when executed by a system having an emulator and atleast one detector, wherein the emulator is configured to be held andoperated in a free working space (FWS), the emulator representing amedical device at a target site, and the at least one detector isconfigured to track the emulator within the FWS, cause the system toperform operations comprising: determining an alignment offset between alocation of the emulator and a location of the medical device; during amedical procedure, determining a first movement amount based on a signalfrom the at least one detector indicative of a first movement of theemulator within the FWS; adjusting the first movement amount by a firstadjustment value; and generating instructions to move the medical devicebased on the adjusted first movement amount, wherein the movement of themedical device by the adjusted first movement amount reduces thealignment offset between the location of the emulator and the locationof the medical device.
 16. The non-transitory computer readable storagemedium of claim 15, wherein the alignment offset comprises a pluralityof adjustment values configured to iteratively reduce the alignmentoffset, the plurality of adjustment values including at least the firstadjustment value and a second adjustment value.
 17. The non-transitorycomputer readable storage medium of claim 15, wherein the firstadjustment value is a function of at least the first movement of theemulator, the alignment offset between the location of the emulator andthe location of the medical device, and the first movement amount. 18.The non-transitory computer readable storage medium of claim 15, theoperations further comprising: during the medical procedure, determininga second movement amount based on a signal from the at least onedetector indicative of a second movement of the emulator within the FWS;adjusting the second movement amount by a second adjustment value; andgenerating instructions to move the medical device based on the adjustedsecond movement amount, wherein the movement of the medical device bythe adjusted second movement amount reduces the alignment offset betweenthe location of the emulator and the location of the medical device. 19.The non-transitory computer readable storage medium of claim 18, whereinthe movement of the medical device by the adjusted first movement amountand the adjusted second movement amount eliminates the alignment offsetbetween the location of the emulator and the location of the medicaldevice.
 20. The non-transitory computer readable storage medium of claim15, the operations further comprising: receiving, from the at least onedetector, first data indicative of at least an orientation of theemulator, the first data comprising roll data, pitch data, and yaw dataof the emulator.